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(57) In an image processing device (10D) for 
processing image data constituting a dynamic image, 
pixels of every frame forming the dynamic image are 
thinned out removing pixels by determining a 5-pixel lat- 
tice format in both the space direction and the time 



direction, thereby reducing the amount of information 
while maintaining the resolutions in the horizontal, verti- 
cal and oblique directions of the dynamic image. 
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Description 

Field of the Invention 

5 [0001] This invention is directed to an image processing apparatus, an image processing method, a transmission 
medium, a transmitting method and an image format, and especially directed to an image processing apparatus, an 
image processing method, a transmission medium, a transmitting method and an image format which make it possible 
to alleviate the deterioration of image quality of a dynamic image. 

10 Description of the Related Art 

[0002] Heretofore, interlace scanning is known as a scanning method for displaying a dynamic image. 
[0003] That is, for instance, in the case where an image is shot by means of a video camera, etc., in essence, it is 
ideal, in terms of the image quality and the resolution, to compose each frame with pixels having the same arrangement 
15 as the pixels (represented by O marks, in Fig. 1) of the built-in photo-electric conversion element such as a CCD 
(charge coupled device) as shown in Fig. 1 , and to perform transmission and recording of it. 

[0004] However, this requires that large amount of information is transmitted and recorded. So, in order to lessen the 
amount of information, image is composed in such a manner that the first picture frame (the first field, in the case of 
interlace scanning) is comprised of only the pixels (represented by O marks, in the same figure) of the even-numbered 
20 lines, negating the pixels (represented by • marks, in the same figure) of the odd-numbered lines, and the next picture 
frame is comprised of only the pixels (represented by • marks, in the same figure) of the odd-numbered lines, negating 
the pixels (represented by O marks, in the same figure) of the even-numbered lines, and the succeeding picture frame 
is composed in the same way as the first picture frame; and these process are repeated in the following, as shown in 
Fig. 2. 

25 [0005] For convenience, such a thinning method is referred to as interlace thinning, since interlace scanning can be 
considered to be one of methods for thinning the pixels for the purpose of reduction of the amount of information as 
known from the above. 

[0006] By the way, the resolution of image in the vertical direction is depends on the number of the lines. Therefore, 
when interlace thinning has been performed, the vertical resolution of the thinned image becomes to 1/2 of the vertical 
30 resolution of the image that is not yet subjected to thinning out. As a result of this, it becomes difficult to represent fast 
changes of the image in vertical direction, and so there is a problem that if such fast movements exist, the image quality 
is deteriorated. 

Summary of the Invention 

35 

[0007] Considering such an aspect, the present invention provides an image processing apparatus, an image 
processing method, a transmission medium, a record medium, and an image format which are able to alleviate the 
deterioration of the image quality that emerges from thinning out. 

[0008] The image processing apparatus and method of the present invention for processing an image data that com- 
40 poses a dynamic image is characterized by thinning the picture elements (pixels) of the respective frames that compose 
the dynamic image, in such a manner that they become a chessboard-like lattice format in both directions of the spatial 
direction and the temporal direction, thereby producing a thinned image data, and then outputting the thinned image 
data. Besides, the image processing apparatus and method of the present invention is further characterized by perform- 
ing bit-thinning of the direction of the level of each pixel that composes the thinned image data, thereby producing a 
45 modified thinned image data, and then outputting the modified thinned image data. 

[0009] The transmitting method and the transmission medium of the present invention for transmitting the data that 
has been produced by thinning the pixels of the image data that composes a dynamic image is characterized by thin- 
ning the pixels of the respective frames that compose the dynamic image, in such a manner that they become a chess- 
board-like lattice format in both directions of the spatial direction and the temporal direction, thereby producing a 
so thinned image data, and then transmitting the thinned image data. Besides, the transmitting method and the transmis- 
sion medium of the present invention is further characterized by performing bit-thinning of the direction of the level of 
each of the pixels that compose the thinned image data, thereby producing a modified thinned image data, and then 
transmitting the modified thinned image data as the transmission data. 

[0010] The image format of composing a dynamic image according to the present invention is characterized by that 
55 the pixels of the respective frames composing a dynamic image have been thinned, in such a manner that they have 
become a chessboard-like lattice format in both directions of the spatial direction and the temporal direction. Further- 
more, the image format according to the present invention is characterized by that bit-thinning of the direction of the 
level of each pixel is performed, with respect to the pixels of the data of which thinning have been performed for each 
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frame. 

[001 1 ] The image processing apparatus according to the present invention for processing the thinned data that has 
been produced by thinning the pixels of the image data composing a dynamic image comprises: a receiving means for 
receiving the thinned image data that has been produced by thinning the pixels of the respective frames that compose 
5 the dynamic image, in such a manner that they become a chessboard-like lattice format in both directions of the spatial 
direction and the temporal direction; and a restoring means for producing the negated pixels, with respect to the thinned 
image data, and for restoring the original image data. 

[001 2] Besides, the image processing apparatus according to the present invention is characterized by that the restor- 
ing means is comprised of a determining means for determining a stated class that represents the property of the atten- 
10 tional negated pixel of the thinned image data, and a generating means for predicting the original pixel in response to 
the determined class and for generating the original pixel data. 

[0013] Besides, the image processing apparatus according to the present invention is characterized by that the gen- 
erating means includes a memory for storing a predictive data, for each class, that has been previously produced for 
each class on the basis of learning by the use of the image data that composes a dynamic image in a standard manner, 
75 reads out a predictive data that corresponds to said determined class, and generates the original pixel data on the basis 
of the predictive data that has been read out. 

[0014] The image processing apparatus according to the present invention for processing the thinned data that has 
been produced by thinning the pixels of the image data composing a dynamic image comprises: a receiving means for 
receiving the modified thinned image data that has been obtained by further performing bit-thinning of the direction of 

20 the level of the pixel, toward each pixel of the thinned image data that has been produced by thinning the pixels of the 
respective frames that compose the dynamic image for each frame, in such a manner that they become a chessboard- 
like lattice format in both directions of the spatial direction and the temporal direction; and a restoring means for produc- 
ing the original pixel with respect to each bit-thinned pixel of the modified thinned image data, thereby producing the 
thinned image data, and for restoring the original image data from the thinned pixels of the thinned image data. 

25 [001 5] Besides, the image processing apparatus according to the present invention is characterized by that the restor- 
ing means is comprised of a first determining means for determining a stated class that represents the property of the 
attentional bit-thinned pixel of the bit-thinned image data, a first generating means for predicting the original pixel in 
response to the determined class and thereby generating the original pixel data, a second determining means for deter- 
mining a stated class that represents the property of the attentional negated pixel of the thinned image data that is com- 

30 posed of said pixel data generated by the first generating means, and a second generating means for predicting the 
original pixel in response to the determined class and for generating the original pixel data. 

[0016] Besides, the image processing apparatus according to the present invention is characterized by that the first 
generating means includes a memory for storing a predictive data, for each class, that has been previously produced 
for each class on the basis of learning by the use of the image data that composes a dynamic image in a standard man- 

35 ner, reads out the predictive data that corresponds to the class that has been determined by said first determining 
means, and generates the original pixel data on the basis of the very predictive data that has been read out, and each 
of the second generating means includes a memory for storing a predictive data, for each class, that has been previ- 
ously produced for each class on the basis of learning by the use of the image data that composes a dynamic image in 
a standard manner, reads out the predictive data that corresponds to the class that has been determined by said sec- 

40 ond determining means, and generates the original pixel data on the basis of the very predictive data that has been 
read out. 

Brief Description of Drawings 

45 [0017] 

Fig. 1 is a diagram showing an image that has been not yet subjected to interlace thinning; 
Fig. 2 is a diagram showing an image that has been subjected to interlace thinning; 

Fig. 3 is a block diagram showing a configuration of the first embodiment of a transmitting/receiving system to which 
\ so the invention has been applied; 

Fig. 4 is a diagram illustrating the processing of a subsampling circuit 2 of Fig. 3; 
Fig. 5 is a diagram illustrating the processing of the subsampling circuit 2 of Fig. 3; 
Fig. 6 is a diagram illustrating the processing of the subsampling circuit 2 of Fig. 3; 

Fig. 7 is a diagram showing a spatial frequency band of an image that is outputted from the subsampling circuit 2 
55 of Fig. 3; 

Fig. 8 is a diagram showing a spatial frequency band of an image that is obtained by performing simple chess- 
board-like pattern thinning; 

Fig. 9 is a diagram illustrating the processing of a blocking circuit 1 1 of Fig. 3; 
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Fig. 10 is a diagram illustrating the processing of a clustering circuit 12 of Fig. 3; 
Fig. 1 1 is a diagram illustrating ADRC that is utilized for clustering; 

Fig. 12 is a block diagram showing a configuration of the first embodiment of a learning apparatus for finding pre- 
dictive data in the first embodiment of the transmitting/receiving system to which the invention has been applied; 
5 Fig. 1 3 is a block diagram showing a configuration of the second embodiment of the learning apparatus for finding 

predictive data in the first embodiment of the transmitting/receiving system to which the invention has been applied; 
Fig. 14 is a diagram illustrating the processing of the blocking circuit 1 1 of Fig. 3; 

Fig. 1 5 is a block diagram showing a configuration of the second embodiment of the transmitting/receiving system 
to which the invention has been applied; 
10 Fig. 1 6 is a block diagram showing a configuration of the third embodiment of the transmitting/receiving system to 
which the invention has been applied; 

Fig. 1 7 is a diagram illustrating the processing of a clustering circuit 53 of Fig. 16; 

Fig. 18 is a block diagram showing a configuration of the first embodiment of a learning apparatus for finding pre- 
dictive data in the second embodiment of the transmitting/receiving system to which the invention has been 
75 applied; and 

Fig. 1 9 is a block diagram showing a configuration of the second embodiment of the learning apparatus for finding 
predictive data in the second embodiment of the transmitting/receiving system to which the invention has been 
applied. 

20 Description of the Preferred Embodiments 

[0018] Referring to the drawings, the embodiments of the present invention are hereinafter described in more detail. 
[0019] Fig. 3 shows a configuration of the first embodiment of a transmitting/receiving system to which the invention 
has been applied ("system" refers to plural devices that have been logically combined and that is not questioned 

25 whether the components exist within the identical housing). 

[0020] This transmitting/receiving system is comprised of a transmitter 10D and a receiver 20D. The transmitter 10D 
is comprised of an input terminal 1 , a subsampling circuit 2, an encoder 3, a transmission processing circuit 4 and an 
output terminal 5, and adapted to compress image data by thinning the pixels and to transmit it. While, the receiver 20D 
is comprised of an input terminal 7, a reception processing circuit 8, a decoder 9, a synthesis circuit 10, a blocking cir- 

30 cuit 11, a clustering circuit 12, a memory 13, an interpolation circuit 14 and an output terminal 15, and adapted to 
expand the compressed image data from the transmitter 10D. 

[0021] That is, the transmitter 10D is supplied with a digital image data of a dynamic image (which may be referred 
to as a whole-pixel dynamic image, hereinafter) wherein each of whose frame is comprised of pixels that have been 
shot with a video camera or the like (not shown) and that have the same arrangement as that of the pixels of an inte- 
35 grated optical-electric conversion element such as a CCD, for instance. In this connection, such a whole-pixel dynamic 
image can be obtained by a so-called progressive camera (a video camera that scans all pixels exist on a photoelectric 
conversion element in an order of so-called raster scanning, and outputs it as an image signal of 1 frame) or a whole- 
pixel camera (a video camera that outputs signals of all pixels exist on a photoelectric conversion element, at a time, as 
an image signal of 1 frame). 

40 [0022] This digital image data is supplied to the subsampling circuit 2, via the input terminal 1 . In the subsampling 
circuit 2, pixels of the respective frames that compose a dynamic image, that is a digital image data, are thinned and 
compressed in such a manner that they become the chessboard-like lattice format in both directions of the spatial direc- 
tion and the temporal direction as shown in Fig. 4 as an example. 

[0023] Both of • marks and O marks are denoting pixels of frames that compose a whole-pixel dynamic image, in 
45 Fig. 4 (as well as the following Fig. 5, Fig. 6. Fig. 9 and Fig. 14). The • marks are denoting the pixels that have been 
negated at the subsampling circuit 2 (these may be referred to as negated pixels (pixel data), hereinafter), while the O 
marks are denoting the pixels that are remaining after thinning has been performed by the subsampling circuit 2 (these 
may be referred to as thinned pixels (pixel data), hereinafter). 

[0024] That is, in the subsampling circuit 2. with respect to the first frame, as shown in Fig. 5 (A) as an example, the 
so pixels of the second column, the fourth column, the sixth column, the eighth column, • • • of the first line, and the first 
column, the third column, the fifth column, the seventh column, • • • of the second line are negated (denoted by • 
marks, in said figure (A)); similarly, with respect to the odd-numbered frames, thinning is performed in the spatial direc- 
tion, in such a manner that the pixels become the chessboard-like lattice pattern. While, with respect to the second 
frame, as shown in Fig. 5 (B) as an example, the pixels of the first column, the third column, the fifth column, the seventh 
55 column, • • • of the first line, and the second column, the fourth column, the sixth column, the eighth column, • • • 
of the second line are negated (denoted by • marks, in said figure (B)); similarly, with respect to the even-numbered 
frames also, thinning is performed in the spatial direction, in such a manner that the pixels become the chessboard-like 
lattice pattern. 
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[0025] Besides, in this case, when the attention is given to a certain line or a certain column, as shown in Fig. 6 as 
an example, with respect to the first frame, the second, the fourth, the sixth, the eighth, • • • pixels that compose the 
very attentional line or column (denoted by • marks, in said figure) are negated, and. with respect to the second frame, 
the first, the third, the fifth, the seventh, • • • pixels that compose the very attentional line or column (denoted by • 

5 marks, in said figure) are negated. And, with respect to the third frame, the second, the fourth, the sixth, the eighth, • 
• • pixels that compose the very attentional line or column (denoted by • marks, in said figure) are once more 
negated, and, thinning is performed for the succeeding frames in the same way; therefore, thinning in the temporal 
direction is also performed in the subsampling circuit 2, into the chessboard-like lattice pattern. 
[0026] With respect to the respective frames of the whole-pixel dynamic image, thinning is performed into the chess- 

w board-like lattice pattern in both directions of the spatial direction and the temporal direction (this may be referred to as 
spatial/temporal chessboard-like pattern thinning, hereinafter) as stated above, so that the alternate pixels are leaved 
in either direction of the spatial direction and the temporal direction. Therefore, the amount of the information that have 
been subjected to spatial/temporal chessboard-like pattern thinning becomes 1/2 of that of the quantity of the original 
information, for the most part, like the foregoing case where the interlace thinning has been performed. 

15 [0027] Fig. 7 shows the spatial frequency band (the portion having slant lines in the figure) of an image that have been 
subjected to spatial/temporal chessboard-like pattern thinning. The spatial frequency band of the image that have been 
subjected to spatial/temporal chessboard-like pattern thinning becomes rectangular, as shown in the figure, as a result 
of it, the resolution of oblique direction is maintained to some extent, as well as the resolution of horizontal direction and 
vertical direction. 

20 [0028] In contrast, Fig. 8 shows the spatial frequency band (the portion having slant lines in the figure) of the image 
that have been subjected to chessboard-like lattice pattern thinning of only spatial direction (chessboard-like lattice pat- 
tern thinning in which pixels that are to be thinned out are the same between the adjacent frames). In this case, the spa- 
tial frequency band becomes rhombus; as a result of it, though the resolutions of horizontal and vertical directions are 
maintained to the original resolution, the resolution of oblique direction lowers to 1/2 of the original. 

25 [0029] Therefore, by performing spatial/temporal chessboard-like pattern thinning, the amount of information can be 
lessen, and besides, every resolutions of horizontal, vertical and oblique directions can be maintained. As a result, it 
becomes possible to represent fast changes in all these directions, and it becomes possible to alleviate (prevent) dete- 
rioration of image quality of the case where such movements exist. However, when the attention is given to a certain 
frame, its resolution of oblique direction lowers even the case of spatial/temporal chessboard-like pattern thinning. 

30 [0030] Returning to Fig. 3, the thinned image data that is comprised of the thinned pixels (pixel data) that have been 
subjected to spatial/temporal chessboard-like pattern thinning at the subsampling circuit 2 is supplied to the encoder 3. 
In the encoder 3. the data from the subsampling circuit 2 (the image data that is comprised of the thinned pixels (pixel 
data)) is subjected to high efficiency coding. As high efficiency coding methods, there are a hybrid system that uses 
orthogonal transformation such as DCT (discrete cosine transform) and motion compensation, a hybrid system that 

35 uses DCT and quantization, a system that uses ADRC (Adaptive Dynamic Range Coding), and others. Suggestions of 
ADRC are disclosed in United States Patent No. 4,703,352 (Issued Date: October 27. 1987) and others. 
[0031] The coded data that is obtained by exposing the thinned image data to high efficiency coding in the encoder 
3 is supplied to the transmission processing circuit 4. The transmission processing circuit 4 exposes the coded data, 
which has been delivered from the encoder 3, to required signal processings such as error correction, packet assem- 

40 bling, channel coding, and outputs the resulted transmission data via the output terminal 5. This transmission data is 
transmitted via a stated transmission line 6. The transmission line 6 can include communication circuits such as a sat- 
ellite circuit, a ground wave, a CATV network, a public network and an internet, and besides, can include a magnetic 
recording/reproducing process, and recording media such as a magnetic disk, an optical disk, a magnetic tape and a 
magneto-optic disk. 

45 [0032] The transmission data from the transmission line 6 is received by the reception processing circuit 8, via the 
input terminal 7 of the receiver 20D. In the reception processing circuit 8, the transmission data is subjected to the 
required signal processings such as channel decoding, packet disassembling and error correction; the coded data that 
is obtained as a result of it is supplied to the decoder 9. In the decoder 9, this coded data is decoded through the decod- 
ing process that corresponds to the coding process of the transmitter 10D, hereby it becomes the thinned image data 
\ so that is comprised of the thinned pixels (pixel data), and then it is supplied to the synthesis circuit 10 and the blocking 
circuit 11. 

[0033] As shown in Fig. 1 1 for instance, the blocking circuit 1 1 groups four thinned pixels (pixel data) a, b, c and d that 
are neighboring the attentional negated pixel x for the stated position (the position that is to be interpolated) upwards, 
downwards, left and right respectively of the spatial direction into one block, so as to generate a class tap and a predic- 
55 tive tap, and then supplies the class tap to the clustering circuit 12 and supplies the predictive tap to the interpolation 
circuit 14 respectively. Toward each negated pixel, the blocking circuit 1 1 composes a block that is comprised of the 
thinned pixels (pixel data) neighboring the negated pixel upwards, downwards, left and right of the spatial direction, and 
then supplies it to the clustering circuit 12 as a class tap, and to the interpolation data producing circuit 14 as a predic- 
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tion tap, respectively. As used herein, the class tap and the prediction tap are the same one. 

[0034] The clustering circuit 1 2 clusters the blocks that have been delivered from the blocking circuit 1 1 into the stated 
class, in accordance with the properties of the thinned pixels that are composing the blocks. 
[0035] At here, a brief explanation is given to clustering. 
s [0036] Supposing that a block (class tap) is comprised of 2 x 2 pixels as shown in Fi. 10 (A) and each of the pixels 
can be represented by 1 bit (it takes either 0 level or 1 level); this block (class tap) of four pixels (2x2 pixels) can be 
classified into 16 (=(2 1 ) 4 ) patterns on the basis of the level distribution of the respective pixels, as shown in Fig. 10 
(B). Classification like this is referred to as clustering, and performed in the clustering circuit 12. 
[0037] Usually, for instance 8 bits are allocated to each pixel (in order to represent the pixel value), in this embodi- 
ed ment, a block (class tap) is comprised of four pixels as stated above. Therefore, if clustering was performed toward such 
blocks (class taps), a very large number of (2 8 ) 4 of classes are created. 

[0038] And so, the clustering circuit 12 exposes the block (class tap) to, for instance, ADRC processing, so that the 
number of bits of the pixels that compose the block can be lessened, and the number of classes can be reduced. In the 
clustering circuit 12 of this embodiment, in fact, one-bit ADRC processing is performed; however, two-bit ADRC 

is processing is described hereinafter, in order to explain ADRC processing. 

[0039] That is, for instance, now, in order to simplify the explanation, as shown in Fig. 1 1 (A), considering a block that 
is comprised of four pixels aligned on a straight line, in ADRC processing, the maximum value MAX and the minimum 
value MIN of the pixel value are detected. And, DR = MAX - MIN is regarded as the local dynamic range of the block, 
and then the pixel value of the pixel that composes the block is re-quantized into K bits, on the basis of this dynamic 

20 range DR. 

[0040] That is, the minimum value MIN is subtracted from each pixel value in the block, and the resulted differences 
are divided by DR/2 K . Then, conversion into the code (ADRC code) that corresponds to the resulted quotient is per- 
formed. To put it concretely, in the case of K = 2 as an example, to which range the quotient belongs out of the ranges 
that are obtained by dividing the dynamic range DR into equal four (= 2 2 ) parts is judged, and when the quotient 
25 belongs to the lowest range, the second range from the bottom, the third range from the bottom, or the highest range, 
coding is performed into 00B, 01 B, 1 0B or 1 1 B respectively of two bits (B designates binary numeral), as shown in Fig. 
11 (B). 

[0041] Decoding of it can be performed by respectively converting the ADRC code 00B t 01B, 10B or 1 1 B into the cen- 
tral value LOO of the lowest level range, the central value L01 of the range of the second level from the bottom, the cen- 
30 tral value L1 0 of the range of the third level from the bottom, or the central value L1 1 of the highest level range, and then 
adding the minimum value MIN to the value that has been obtained by said conversion. 

[0042] In this connection, clustering can be performed on the basis of, for instance, tendency of levels of pixels that 
compose a block (class tap) (for instance, such a tendency that the levels of all pixels are nearly uniform, or that the 
levels of the righthand pixels are higher or lower than the levels of the other pixels), etc., other than performing on the 

35 basis of the level itself of the respective pixels that compose the block (class tap) as stated above. 

[0043] Returning to Fig. 3 once more, the class that has been obtained by said clustering is given to the memory 13 
as an address. The memory 13 is storing the predictive data, for each class, for predicting the negated pixel x that has 
been thinned out from the pixels composing the whole-pixel dynamic image, and, when it has received the class as the 
address from the clustering circuit 12, it reads the predictive data that corresponds to the class and then supplies to the 

40 interpolation circuit 1 4. 

[0044] In this case, the memory 13 is storing, for instance, a coefficient set w1 ,. w2, w3 and w4 of a linear expression 
that is using the thinned pixels (pixel data) a - d (Fig. 9) (predictive tap) that are neighboring the attentional negated pixel 
x for the stated position (the position that is to be interpolated) upwards, downwards, left and right respectively, as the 
predictive data; and the linear expression is used to predict the attentional negated pixel x. Therefore, from the memory 
45 13, the coefficient set w1 - w4 that corresponds to the class for the attentional negated pixel is supplied to the interpo- 
lation circuit 14 as the predictive data. 

[0045] Receiving the coefficient set wl , w2, w3 and w4 as the predictive data, the interpolation data producing circuit 
14 produces (predicts) the attentional negated pixel (pixel data) x, by calculating the following linear expression, using 
the coefficient set w1 - w4, which is treated as the predictive data, and the pixel values a - d of the thinned pixels (pixel 
so data) that compose the block (predictive tap) supplied from the blocking circuit 1 1 . 

x = w1a + w2b + w3c + w4d (1) 

[0046] The attentional negated pixel (pixel data) x that has been obtained in the interpolation data producing circuit 
55 14 is outputted to the synthesis circuit 10. At the timing for outputting the thinned pixels, the synthesis circuit 10 selects 
the thinned pixels (pixel data) supplied from the decoder 9, while, at the timing for outputting the negated pixels, the cir- 
cuit 1 0 selects the negated pixels (pixel data) supplied from the interpolation data producing circuit 1 4, and then outputs 
them via the output terminal 15. As a result of this, the data of the frames that compose the whole-pixel dynamic image 
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is outputted from the output terminal 15. 

[0047] Next, for the purpose of obtaining the negated pixel (pixel data), the coefficient set w1 , w2, w3 and w4, which 

is the predictive data for composing the linear expression of Equation (1), is found by learning. 

[0048] Fig. 12 shows an example of configuration of an embodiment of a learning apparatus that performs learning 

5 for finding the coefficient set w1 , w2, w3 and w4 as the predictive data. 

[0049] A whole-pixel dynamic image that is to be used in learning is inputted to an input terminal 21, in units of a 
frame, for instance. In this connection, it is desirable that the whole-pixel dynamic image for learning is a standard one 
in which production of the coefficient set w1 , w2, w3 and w4 of the predictive data has been considered. 
[0050] A frame of the whole-pixel dynamic image that has been inputted to the input terminal 21 is supplied to a block- 

io ing circuit 22. Treating a pixel (pixel data) that is the attentional negated pixel x shown in Fig. 9. as the attentional pixel 
(pixel data), the blocking circuit 22 selects the pixels (pixel data) that are regarded as the thinned and leaved pixels a - 
d in relation to the attentional pixel (pixel data), out of the pixels (pixel data) that are composing the supplied frame; then, 
the blocking circuit 22 forms a block (a class tap and a predictive tap) with these four pixels (pixel data), and supplies it 
to a clustering circuit 24. Besides, the blocking circuit 22 supplies five pixels (pixel data) that is the sum of four pixels 

is (pixel data) composing the block and the attentional pixel (pixel data) x to an input terminal IN of a data memory 23. 
[0051] In the same way as the clustering circuit 12 of Fig. 3, the clustering circuit 24 clusters the blocks (class taps) 
that are supplied to the circuit 24, and supplies the resulted class to a terminal 25a of a switch 25. The switch 25 is 
selecting the terminal 25a, until clustering have been finished with respect to all of blocks that are obtained from the 
whole-pixel dynamic image for learning; therefore, the classes in relation to the respective attentional negated pixels 

20 that are outputted from the clustering circuit 24 are supplied to the address terminal AD of the data memory 23, via the 
switch 25. 

[0052] The data memory 23 stores the image data that is supplied to its input terminal IN, into the address corre- 
sponding to the class that is supplied to its address terminal AD. 

[0053] At here, x1, x2, • • xn denote the attentional pixels (pixel data) related to blocks that are classified into the 
25 stated class Class, out of all blocks (class taps) that are obtained from the whole-pixel dynamic image for learning. 
Besides, the pixels (pixel data) that are neighboring the attentional pixel (pixel data) x1 upwards, downwards, left and 
right respectively are designated by a1 , b1 , c1 and d1 ; the pixels (pixel data) that are neighboring the attentional pixel 
(pixel data) x2 upwards, downwards, left and right respectively are designated by a2, b2, c2 and d2; • • • ; the pixels 
(pixel data) that are neighboring the attentional pixel (pixel data) xn upwards, downwards, left and right respectively are 
30 designated by an, bn, cn and dn. In this case, by the above-mentioned processing, x1 , x2, • • • , xn, a1,a2, • • • , 
an,b1,b2, • • • ,bn,c1,c2, • • • , cn, anddl, d2, • • • , dn are stored in the address that corresponds to the class 
Class in the memory 23 as the respective pixel data. 

[0054] When clustering have been completed with respect to all blocks (class taps) that are obtained from the whole- 
pixel dynamic image for learning, the switch 25 selects a terminal 25b. The terminal 25b is adapted to receive the output 
35 of a counter 26, and the counter 26 is adapted to generate addresses that sequentially change, by counting the stated 
clock CK. So, the addresses that are generated in the counter 26 are outputted via the switch 25. 
[0055] The addresses that are outputted from the counter 26 via the switch 25 are supplied to the address terminal 
AD of the data memory 23 and to the address terminal AD of a memory 28. 

[0056] The data memory 23 reads out its storing content (attentional pixel (pixel data)) in accordance with the address 
40 that has been supplied to the address terminal AD from the counter 26, and also reads out four pixels (pixel data) that 
are neighboring it, and then supplies them to an arithmetic circuit of the method of least squares 27. On the basis of the 
data that is supplied from the data memory 23, the arithmetic circuit of the method of least squares 27 establishes equa- 
tions, and then solves the equations with the method of least squares for instance, so as to find the coefficient set w1 - 
w4 that is treated as the predictive data. 
45 [0057] That is, in the case where a notice has been given to the above-mentioned class Class, the arithmetic circuit 
of the method of least squares 27 makes the following simultaneous equations that are correspond to the equation (1), 
using the data x1, x2, • • • ,xn, a1,a2, • • • , an, b1,b2, • • • , bn, c1,c2, • • • , cn, anddl, d2, • • • ,dnthat 
have been stored in the address corresponding to the class Class of the data memory 23. 

\ 50 



55 



7 



EP0 920 215 A1 



xl 



wlal + w2bl + w3cl + w4dl 



x2 = wla2 + w2b2 + w3c2 + w4d2 



5 



70 



xn 



wlan + w2bn + w3cn + w4dn 



15 



(2) 



[0058] And then, the arithmetic circuit of the method of least squares 27 solves the simultaneous equations of the 
20 equation (2) by the method of least squares, and hereby finds the coefficient set wl - w4 that is treated as the predictive 

data of the class Class. The predictive data related to the other classes can be found in like manner. 

[0059] The coefficient set w1 - w4, which is treated as the predictive data, that has been found in the arithmetic circuit 

of the method of least squares 27 is supplied to a memory 28. Therefore, the coefficient set w1 - w4, which is treated 

as the predictive data related to the class Class, is stored in the same address of the memory 28 as the address of the 
25 memory 23 in which the data x1, x2, • • • .xn.a1.a2, • • • ,an,b1,b2, • • • .bn.c1.c2, • • • , cn, and d1, d2. 

• • • , dn is being stored. The coefficient sets that are treated as the predictive data related to the other classes are 

stored in the memory 28 in like manner. 

[0060] In the memory 13 of Fig. 13, the predictive data that have been stored in the memory 28 in this manner are 
stored. 

30 [0061] Next, into the memory 13 of Fig. 3, the pixel value itself can be stored as the predictive data, other than the 
coefficient set for use in calculation of the linear expression shown in the equation (1). 

[0062] Fig. 1 3 shows an example of configuration of the second embodiment of the learning apparatus for finding the 
predictive data, of the case where the pixel value is stored in the memory 13 as the predictive data. In this figure, the 
parts that correspond to the parts in Fig. 12 have the same reference numerals. 

35 [0063] The whole-pixel dynamic image that is to be used in learning is inputted to the input terminal 21, and the 
frames of the whole-pixel dynamic image are supplied to the blocking circuit 22. From the pixels that are composing the 
supplied frame, the blocking circuit 22 forms blocks (class taps) in the same way as the case of Fig. 12, and supplies 
them to the clustering circuit 24. Besides, the blocking circuit 22 supplies only attentional pixels (pixel data) that become 
the attentional negated pixels x to a computing element 34. 

40 [0064] The clustering circuit 24 clusters the supplied blocks (class taps) in the same way as the clustering circuit 1 2 
of Fig. 3, and supplies the resulted class to the address terminal AD of the data memory 30 and to the address terminal 
AD of a frequency memory 31 . 

[0065] Before learning is commenced, the stored contents of the data memory 30 and the frequency memory 31 are 
cleared to 0. 

45 [0066] When the class that is treated as an address has been supplied to the address terminal AD of the frequency 
memory 31 , the frequency that is the stored contents of the very address is read out, and outputted from the output ter- 
minal OUT of the frequency memory 31 . The frequency that has been outputted from the frequency memory 31 is sup- 
plied to a computing element 32, and increased by 1 . The result of this incrementing is supplied to the input terminal IN 
of the frequency memory 31 , and stored (overwritten) in the address in which the frequency that is not yet incremented 

so has been being stored. ? * 

[0067] As to the data memory 30, when the class that is treated as an address has been supplied to its address ter- 
minal AD, the stored contents of the very address is read out too, and outputted from its output terminal OUT. The out- 
put of the data memory 30 is supplied to a computing element 33. The frequency that is outputted by the frequency 
memory 31 is also supplied to the computing element 33, and multiplication of this frequency by the output of the data 

55 memory 30 is performed at there. The result of this multiplication is supplied to a computing element 34. 

[0068] In the computing element 34, the resulted product of the multiplication that has been performed in the comput- 
ing element 33 is added to the pixel value of the attentional pixel (pixel data) that has been delivered from the blocking 
circuit 22. and the obtained sum value is supplied to a computing element 35. In addition, the result of incrementing of 
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the frequency that has been performed by the computing element 32 is also supplied to the computing element 35, and 
a division is performed at there, treating the result of addition that has been performed in the computing element 34 as 
the dividend, and treating the result of incrementing that has been performed in the adder 32 as the divisor The result 
of this division is supplied to the input terminal IN of the data memory 30, and stored (overwritten) in the address cor- 

5 responding to the class that is outputted from the clustering circuit 24. 

[0069] In the case of the first accessing to a certain address ad of the data memory 30 and the frequency memory 
31 in the learning apparatus of Fig. 13, the data x1 that is supplied to the computing element 34 from the blocking circuit 
22 is written in the address ad of the data memory 30 as it is, and 1 is written in the address ad of the frequency memory 
31 . Supposing that accessing to the address ad is subsequently performed again and the data that is supplied to the 

10 computing element 34 from the blocking circuit 22 is x2, the output of the computing element 32 becomes 2, and the 
output of the computing element 34 becomes x1 + x2; therefore, the output of the computing element 35 becomes (x1 
+ x2)/2, and this value is written into the address ad of the data memory 30. And, into the address ad of the frequency 
memory 31. the value 2 that is the output of the computing element 32 is written. Supposing that accessing to the 
address ad is performed again and the data that is supplied to the computing element 34 from the blocking circuit 22 is 

is x3. the value (x1 + x2 + x3)/3 is written into the address ad of the data memory 30. and the value 3 is written into the 
address ad of the frequency memory 31 . 

[0070] In this way, the mean value of the pixel values of the attentional pixels (pixel data) that are classified into each 
class is stored in the data memory 30. 

[0071 ] In the case where the pixel values that have been stored in the data memory 30 is stored in the memory 1 3 of 
20 Fig. 3 as the data for prediction, reading the pixel value that is regarded as the predictive data from the memory 13 is 
predicting the negated pixel, therefore, it is not needed to provide the interpolation data producing circuit 14. 
[0072] By the way, in the above-mentioned case, the blocking circuit 1 1 of Fig. 3 forms one block (a class tap and a 
predictive tap) with four thinned pixels (pixel data) a, b, c and d that are neighboring the attentional negated pixel x 
upwards, downwards, left and right respectively of the spatial direction, as shown in Fig. 9; however, it is able to form a 
25 block such that the block includes also the thinned pixels that are neighboring the attentional negated pixel x in the tem- 
poral direction. 

[0073] That is, in the case of forming a block with respect to a certain attentional negated pixel x of the n-th frame, it 
is able to form the block (a class tap and a predictive tap) with six pixels (pixel data) that is the total of four thinned pixels 
(pixel data) a, b, c and d neighboring upwards, downwards, left and right respectively in the identical frame, the thinned 
30 pixel (pixel data) e that is resting on the same position of the (n-1)th frame as the position of the attentional negated 
pixel x, and the thinned pixel (pixel data) f that is resting on the same position of the (n+1)th frame as the position of the 
attentional negated pixel x, as shown in Fig. 14. 

[0074] In this case, it is able to restore the negated pixel x such that it becomes closer to the original, because clus- 
tering and calculation of the equation (1) are performed considering the thinned pixels (pixel data) e and f of the tem- 
35 poral direction. In this case, it is needed to form a block in like manner at the time of learning also. Besides, a block (a 
class tap and a predictive tap) can be composed of only thinned pixels that are located in the temporal direction of the 
attentional negated pixel x. 

[0075] In this embodiment, when the notice has been given to a certain negated pixel, the thinned pixels that are used 
for clustering and the thinned pixels that are used for calculating the linear expression shown in the equation (1) are 
40 identical, however, these are not always required to be the identical pixels. That is, it is able to use the distinct sets of 
thinned pixels, for clustering and for calculation of the equation (1). Besides, the method for clipping a class tap or a 
predictive tap may be changed adaptively, in accordance with the spatial feature (activity), movement, and others of the 
image. 

[0076] Besides, thinned pixels that are used for clustering and thinned pixels that are used for calculating the linear 
45 expression shown in the equation (1) are not required to neighbor the negated pixel spatially or temporally. But, it is 
desirable to use the thinned pixels that are resting on the periphery of the negated pixel. 

[0077] Besides, ADRC processing has been used in the clustering circuit, and the class has been determined on the 
basis of the spatial activity, however, the movement of the block that has been detected may be used as the class, or 
the class may be determined based on the combination of ADRC processing and movement detection. 
so [0078] In addition, only negated pixels have been produced in this embodiment, however, thinned pixels may be newly 
produced in like manner. In this case, the synthesis circuit 10 is not needed anymore. 

[0079] In this embodiment, each pixel of the respective frames that compose a dynamic image was 8 bits, however, 
we do not intend to limit the present invention to such; each pixel may be comprised of more bits than 8 bits such as 10 
bits and 12 bits, and also may be comprised of less bits than 8 bits. 
55 [0080] In the above receiver 20D, a negated pixel is predicted using the predictive data that has been obtained by 
learning, so that it is able to restore the high frequency components that are not included within the image that is com- 
prised of the thinned pixels; however, restoring the thinned pixel can be performed by a simple interpolation too. 
[0081] Fig. 15 shows an example of a configuration of the second embodiment of such a transmitting/receiving sys- 
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tern. In this figure, the parts that correspond to the parts in Fig. 1 have the same reference numerals. That is, this trans- 
mitting/receiving system is composed like the transmitting/receiving system of Fig. 3, excepting that an interpolation 
filter 40 is provided in place of the blocking circuit 1 1 , the clustering circuit 12, the memory 13, and the interpolation data 
producing circuit 14. 

5 [0082] In the interpolation filter 40, out of the thinned pixels (pixel data) that have been delivered from the decoder 9. 
the mean value of the thinned pixels (pixel data) that are resting on the periphery of the negated pixel is found, and then 
this mean value is outputted to the synthesis circuit 10 as the interpolation value of the negated pixel (pixel data). 
[0083] In this case, it is not able to restore the high frequency components that are not included within the image data 
that is comprised of the thinned pixels (pixel data) like the case of Fig. 3; however, it is able to simplify the configuration 

10 of the receiver 20D. 

[0084] Next, an explanation is given to a configuration of the third embodiment of the transmitting/receiving system to 
which the invention has been applied. Fig. 16 shows an example of the configuration of the transmitting/receiving sys- 
tem of the third embodiment. In this figure, the parts that correspond to the parts in Fig. 3 have the same reference 
numerals. 

75 [0085] This transmitting/receiving system of the third embodiment is comprised of a transmitter 10D and a receiver 
20D. And, the transmitter 10D is comprised of an input terminal 1, a subsampling circuit 2, a bit lowering circuit 51 , an 
encoder 3, a transmission processing circuit 4 and an output terminal 5, and capable of thinning the pixels of the image 
data, lowering the bits of the thinned pixels with respect to the direction of the level thereby compressing them, and 
transmitting them; while, the receiver 20D is comprised of an input terminal 7, a reception processing circuit 8, a 

20 decoder 9, a blocking circuit 52, a clustering circuit 53, a memory 54, a pixel data producing circuit 55, an image mem- 
ory 56, a synthesis circuit 10, a blocking circuit 11 , a clustering circuit 12, a memory 13, an interpolation data producing 
circuit 14 and an output terminal 15, and capable of expanding the compressed image data that is delivered from the 
transmitter 10D. 

[0086] That is, the transmitter 1 0D is supplied with a digital image data of the dynamic image (which may be referred 
25 to as a whole-pixel dynamic image, hereinafter), in the same way as the first embodiment shown in Fig. 3. It is assumed 
that each pixel that is composing this digital image data is 1 2 bits. This digital image data is supplied to the subsampling 
circuit 2, via the input terminal 1 . In the subsampling circuit 2, in the same way as the first embodiment shown in Fig. 3, 
pixels of the respective frames that compose a dynamic image, that is a digital image data, are thinned and compressed 
in such a manner that they become the chessboard-like lattice pattern in both directions of the spatial direction and the 
30 temporal direction as shown in Fig. 4 as an example. That is, with respect to the respective frames of the whole-pixel 
dynamic image, thinning is performed into the chessboard-like lattice pattern in both directions of the spatial direction 
and the temporal direction (this may be referred to as spatial/temporal chessboard-like pattern thinning, hereinafter), so 
that the thinned image data is produced in which the alternate pixels are leaved in either direction of the Spatial direc- 
tion and the temporal direction. The thinned image data that is comprised of the thinned pixels (pixel data) that have 
35 been subjected to spatial/temporal chessboard-like pattern thinning at the subsampling circuit 2 is supplied to the bit 
lowering circuit 51 . 

[0087] The bit lowering circuit 51 eliminates the bits of 4 bits from LSB (least significant bit) out of 1 2 bits of each pixel 
(pixel data) of the supplied thinned image data, and produces a bit modification pixel (pixel data) of 8 bits. Then, the 
modified thinned image data that is comprised of the respective bit modification pixels (pixel data) is supplied to the 
40 encoder 3. 

[0088] In the encoder 3, the modified thinned image data is subjected to high efficiency coding, and supplied to the 
transmission processing circuit 4 as the coded data, as with the first embodiment shown in Fig. 3. The transmission 
processing circuit 4 exposes the coded data to required signal processings such as error correction, arranging into 
packets, and channel coding, and outputs the resulted transmission data via the output terminal 5, as with the first 

45 embodiment shown in Fig. 3. This transmission data is transmitted via the stated transmission line 6, in like manner. 
[0089] The transmission data from the transmission line 6 is received by the reception processing circuit 8, via the 
input terminal 7 of the receiver 20D. In the reception processing circuit 8, the transmission data is subjected to the 
required signal processings such as channel decoding, packet disassembling, and error correction; the coded data that 
is obtained as a result of it is supplied to the decoder 9. In the decoder 9, this coded data is decoded through the decod- 

50 ing process that corresponds to the coding process of the encoder 3 of the transmitter 1 0D, hereby it becomes the mod- 
ified thinned image data that is comprised of the bit modification pixels (pixel data), and then it is supplied to the blocking 
circuit 52. 

[0090] As shown in Fig. 17 for instance, the blocking circuit 52 groups eight bit modification pixels (pixel data) A, B, 
C, D, E, F, G and H that are neighboring the attentional bit modification pixel x upwards, downwards, left, right and 
55 obliquely of the spatial direction into one block, to generate a class tap and a predictive tap, and then supplies the class 
tap to the clustering circuit 53 and supplies the predictive tap to the pixel data producing circuit 55, respectively. In rela- 
tion to each bit modification pixel, the blocking circuit 52 composes a block that is comprised of the bit modification pixel 
(pixel data) neighboring the bit modification pixel upwards, downwards, left, right and obliquely, and then supplies it to 
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the clustering circuit 53 as a class tap. and to the interpolation data producing circuit 55 as a prediction tap, respectively. 
As used herein, the class tap and the prediction tap are the same one. 

[0091 ] The clustering circuit 53 clusters the blocks that have been delivered from the blocking circuit 52 into the stated 
class, in accordance with the properties of the bit modification pixels that are composing the blocks. The principle of 
5 clustering has been described using Fig. 10 and Fig. 11 at the time of description of the first embodiment, so the 
description of it is omitted here. 

[0092] The clustering circuit 53 exposes eight bit-modification pixels (pixel data) that are composing the block (class 
tap) to 1-bit ADRC processing, thereby generating a 8-bit class code that represents the class concerning the atten- 
tional bit modification pixel x' . 

10 [0093] The class that has been obtained by clustering like the above is given to the memory 54 as an address. The 
memory 54 is storing the predictive data for predicting a 12-bit pixel from a bit modification pixel x that has been modi- 
fied to 8 bits, for each class, and, when the class that is regarded as an address has been given from the clustering 
circuit 53, reads the predictive data that corresponds to that class, and supplies it to the pixel data producing circuit 55. 
[0094] In the present case, the memory 54 is storing the coefficient set w1 , w2, w3, w4, w5, w6, w7 and w8 of a linear 

75 expression for predicting the attentional bit modification pixel x ' of the stated position by use of the linear expression 
that is using the bit modification pixels (pixel data) A - H (Fig. 1 7)(predictive tap) that are neighboring the attentional bit 
modification pixel x' upwards, downwards, left and right respectively, as the predictive data. Therefore, from the memory 
55 to the pixel data producing circuit 55, the coefficient set w1 - w8 that is corresponding to the class concerning the 
attentional bit modification pixel is supplied as the predictive data. 

20 [0095] When a coefficient set w1 - w8 that is regarded as the predictive data has been received, the pixel data pro- 
ducing circuit 55 calculates the following linear expression by the use of the coefficient set w1 - w8 that is regarded as 
the predictive data and the pixel values A - H of the bit modification pixels (pixel data) comprising the block (predictive 
tap) that has been supplied from the blocking circuit 1 1 , and hereby produces (predicts) 12-bit restored pixel (pixel data) 
concerning 8-bit attentional bit modification pixel x. 

25 

x = w1A + w2B + w3C • • • +w8H (3) 

[0096] The pixel x* that has been obtained in the pixel data producing circuit 55 is supplied to the image memory 56 
and stored therein. The above-mentioned processing is performed in relation to, for instance, 1 frame, and the pixel data 
30 that has been respectively produced is stored in the image memory. The image data that is comprised of the pixels 
stored in the image memory 56 is restored as the thinned image data that is nearly the same as the thinned image data 
that is the output of the subsampling circuit 2 of the transmitter 10D. 

[0097] The thinned image data that has been stored in the image memory 56 is the same as the thinned image data 
that has been outputted from the decoder 9 of the receiver 20D of the first embodiment shown in Fig. 3; in the third 

35 embodiment too, the negated pixels are produced in relation to the thinned image data that has been stored in the 
image memory 56, and the image data is restored, in the same way as the first embodiment shown in Fig. 3. As a result 
of this, from the output terminal 15, the data of the frames that compose the whole-pixel dynamic image is outputted. 
The configurations and the operations of the blocking circuit 1 1 , the clustering circuit 12, the memory 13, the interpola- 
tion data producing circuit 14 and the synthesis circuit 10 of the receiver 20D of the third embodiment shown in Fig. 16 

40 are the same as those of the first embodiment, so the explanation is omitted here. 

[0098] Next, to obtain 12-bit restored pixel (pixel data) x* from 8-bit bit modification pixel (pixel data), the set of coef- 
ficients w1 , w2, w3, w4, w5, w6, w7 and w8 that is regarded as the predictive data for comprising the linear expression 
of the equation (3) is obtained by learning. 

[0099] Fig. 18 shows an example of the configuration of the first embodiment of the learning apparatus that performs 
45 learning for finding the set of coefficients w1 , w2, w3, w4, w5. w6, w7 and w8 that is regarded as the predictive data for 
finding the 12-bit restored pixel (pixel data) x' from the 8-bit bit modification pixel (pixel data) in the third embodiment. 
[0100] A whole-pixel dynamic image for learning, each pixel of which is 12 bits, is inputted to an input terminal 21 , in 
units of a frame, for instance. In this connection, it is desirable that the whole-pixel dynamic image for learning is a 
standard one in which production of the coefficient set w1 , w2, w3, w4, w5, w6, w7 and w8 of the predictive data has 
so been considered. 

[0101] A frame of the whole-pixel dynamic image that has been inputted to the input terminal 61 is supplied to a block- 
ing and bit-lowering circuit 62. Treating a 12-bit pixel of the stated attentional bit modification pixel x' shown in Fig. 17 
as the attentional pixel (pixel data), the blocking and bit-lowering circuit 62 selects the 12-bit pixels (pixel data) that are 
regarded as the bit modification pixels (pixel data) A - H in relation to the attentional pixel, out of the pixels that are com- 
55 posing the supplied frame. Then, the blocking circuit 22 eliminates the bits that are 4 bits from LSB (least significant bit), 
out of 12 bits of each pixel of the selected eight pixels (pixel data), to produce a 8-bit bit modification pixel (pixel data). 
Then, it forms a block (a class tap and a predictive tap) with the produced eight bit modification pixels (pixel data), and 
supplies it to a clustering circuit 64. Besides, the blocking and bit-lowering circuit 62 supplies nine pixels (pixel data) that 
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is the sum of the eight bit-modification pixels (pixel data) that are composing the block and the attentional pixel (pixel 
data) x to an input terminal IN of a data memory 23. 

[01 02] In the same way as the clustering circuit 53 of Fig. 1 6, the clustering circuit 64 clusters the blocks (class taps) 
that are supplied thereto, and supplies the resulted class to a terminal 65a of a switch 65. The switch 65 is selecting the 
5 terminal 65a, until clustering have been finished with respect to all of blocks that are obtained from the whole-pixel 
dynamic image for ieaFriing; therefore, the classes initiation to the respective attentional bit-modification pixels that are 
outputted from the clustering circuit 64 are supplied to the address terminal AD of the data memory 63, via the switch 
65. 

[0103] The data memory 63 stores the image data that is supplied to its input terminal IN, into the address corre- 

io sponding to the class that is supplied to its address terminal AD. 

[0104] At here, the attentional pixels (pixel data) related to blocks that are classified into the stated class Class, out 
of ail blocks (class taps) that are obtained from the whole-pixel dynamic image for learning, are denoted by x 1 1 , x'2, • 
* • , x ' n. Besides, the bit modification pixels (pixel data) that are neighboring the attentional pixel x1 upwards, down- 
wards, left and right are denoted by A1 , B1 , C1 , D1 , E1 , F1 , G1 AND H1 ; the bit modification pixels (pixel data) that are 

is neighboring the attentional pixel (pixel data) x2 upwards, downwards, left and right are denoted by A2, B2, C2, D2, E2, 
F2, G2 and H2; • • • ; the bit modification pixels (pixel data) that are neighboring the attentional pixel (pixel data) xn 
upwards, downwards, left and right are denoted by An, Bn, Cn, Dn, En, Fn, Gn and Hn. In this case, by the above-men- 
tioned processing, x*1, x'2, • • • , x 1 n, A1, A2, • • • ,An,B1,B2, • • • , Bn, C1, C2, • • • , Cn, D1,D2, • • • , 
Dn, E1.E2, ■ • -,En, F1.F2, • • • , Fn, G1, G2, • • • , Gn, and H1, H2, • • Hn are stored in the address that 

20 corresponds to the class Class in the memory 23 as the respective pixel data. 

[0105] When clustering have been completed with respect to all blocks (class taps) that are obtained from the whole- 
pixel dynamic image for learning, the switch 65 selects a terminal 65b. The terminal 65b is supplied with the output of 
a counter 66, and the counter 66 is adapted to generate addresses that sequentially change, by counting the stated 
clock CK. So, the addresses that are generated by the counter 66 are outputted via the switch 65. 

25 [0106] The addresses that are outputted from the counter 66 via the switch 65 are supplied to the address terminal 
AD of the data memory 63 and to the address terminal AD of a memory 68. 

[0107] The data memory 63 reads out its storing content (attentional pixel (pixel data)) and the neighboring bit-mod- 
ification pixels (pixel data), in accordance with the address that has been supplied to its address terminal AD from the 
counter 66, and then supplies them to an arithmetic circuit of the method of least squares 67. On the basis of the data 
30 that is supplied from the data memory 63, the arithmetic circuit of the method of least squares 67 establishes equations, 
and then solves the equations with the method of least squares for instance, so as to find the coefficient set w1 - w8 
that is regarded as the predictive data. 

[0108] That is, in the case where a notice has been given to the above-mentioned class Class, the arithmetic circuit 
of the method of least squares 27 establishes the following simultaneous equations that are correspond to the equation 
35 (3), using the data x '1 , x'2, • • • , x 'n, A1, A2, • • •,An,B1,B2, • • •,Bn,C1,C2, • • •,Cn,D1,D2, • • •, 
Dn, E1.E2, • • • ,En, F1.F2, • • • , Fn, G1, G2, • • • , Gn, and H1, H2, • • • , Hn that have been stored in the 
address corresponding to the class Class of the data memory 23. 
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X ' 1 = wlAl + w2Bl + w3Cl + w4Dl + w5El + w6Fl + w7Gl + 

W8H1 

5 

x ' 2 = W1A2 + w2B2 + w3C2 + w4D2 + w5E2 + w6F2 + w7G2 + 

w8H2 

10 



x ' n « wlAn + w2Bn + w3Cn + w4Dn ♦ w5£n + w6Fn + w7Gn + 

w8Hn 

20 

(4) 



25 [0109] And then, the arithmetic circuit of the method of least squares 67 solves the simultaneous equations of the 
equation (4) by the method of least squares, and hereby finds the coefficient set w1 - w8 that is regarded as the predic- 
tive data of the class Class. The predictive data related to the other classes can be found in like manner. 
[0110] The coefficient set w1 - w4, which is regarded as the predictive data, that has been found in the arithmetic 
circuit of the method of least squares 67 is supplied to the memory 68. Therefore, the coefficient set w1 - w4, which is 

30 regarded as the predictive data related to the class Class, is stored in the same address of the memory 68 as the 
address of the memory 63 in which the data x f 1 , x* 2, • • • , x ' n, A1 , A2, • • • , An, B1 , B2, • • • , Bn, CI , C2 f • 
• • , Cn, D1, D2, • • • , Dn, E1, E2, • • • , En, F1, F2, • • • , Fn, G1, G2, • • • , Gn, and H1, H2, • • • , Hn 
are being stored. The coefficient sets that are regarded as the predictive data related to the other classes are stored in 
the memory 28 in like manner. 

35 [0111] In the memory 54 of Fig. 16, the predictive data that have been stored in the memory 68 in this manner are 
stored. As to the memory 13 of Fig. 16, the coefficient set that is regarded as the predictive data and that has been pro- 
duced by the learning apparatus of Fig. 12 is stored therein. 

[0112] Next, into the memory 54 of Fig. 16, the pixel value itself can be stored as the predictive data, other than the 
coefficient set for use in calculation of the linear expression shown in the equation (3). 
40 [01 1 3] Fig. 1 9 shows an example of configuration of the second embodiment of the learning apparatus for finding the 
predictive data, of the case where the pixel value is stored in the memory 54 as the predictive data. In this figure, the 
parts that correspond to the parts in Fig. 18 have the same reference numerals. 

[0114] A frame of the whole-pixel dynamic image that has been inputted to the input terminal 61 is supplied to the 
blocking and bit-lowering circuit 62. From the pixels that are composing the supplied frame, the blocking and bit-lower- 
45 ing circuit 62 forms blocks (class taps) in the same way as the case of Fig. 1 8, and supplies them to the clustering circuit 
64. And, it supplies only 12-bit attentional pixels (pixel data) related to the stated attentional bit-modification pixel x' to 
a computing element 74. 

[01 1 5] The clustering circuit 64 clusters the supplied blocks (class taps) in the same way as the clustering circuit 53 
of Fig. 16, and supplies the resulted class to the address terminal AD of the data memory 70 and to the address termi- 
50 nal AD of the frequency memory 71 . 

[01 1 6] Before learning is commenced, the stored contents of the data memory 70 and the frequency memory 71 are 
cleared to 0. 

[01 1 7] When the class that is regarded as an address has been supplied to the address terminal AD of the frequency 
memory 71 , the frequency that is the stored contents of the very address is read out and outputted from the output ter- 
ss minal OUT of the frequency memory 71 . The frequency that has been outputted from the frequency memory 71 is sup- 
plied to a computing element 72, and increased by 1 . The result of this incrementing is supplied to the input terminal IN 
of the frequency memory 71 , and stored (overwritten) in the address in which the frequency that is not yet incremented 
has been being stored. 
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[0118] As to the data memory 70, when the class that is regarded as an address has been supplied to its address 
terminal AD, the stored contents of the very address is read out similarly, and outputted from its output terminal OUT. 
The output of the data memory 70 is supplied to a computing element 73. The frequency that is outputted by the fre- 
quency memory 71 is also supplied to the computing element 73, and multiplication of this frequency by the output of 

5 the data memory 70 is performed at there. The result of this multiplication is supplied to a computing element 74. 

[01 1 9] In the computing element 74, the resulted product of the multiplication that has been performed in the comput- 
ing element 73 is added to the pixel value of the attentional pixel (pixel data) that has been delivered from the blocking 
and bit-lowering circuit 62, and the obtained sum value is supplied to a computing element 75. In addition, the result of 
incrementing of the frequency that has been performed by the computing element 72 is also supplied to the computing 

10 element 75, and a division is performed at there, treating the result of addition that has been performed in the computing 
element 74 as the dividend, and treating the result of incrementing that has been performed in the adder 72 as the divi- 
sor. The result of this division is supplied to the input terminal IN of the data memory 70, and stored (overwritten) in the 
address corresponding to the class that is outputted from the clustering circuit 74. 

[0120] In the case of the first accessing to a certain address ad of the data memory 70 and the frequency memory 
is 71 in the learning apparatus of Fig. 1 9, the data x'1 that is supplied to the computing element 74 from the blocking and 
bit-lowering circuit 62 is written in the address ad of the data memory 60 as it is, and 1 is written in the address ad of 
the frequency memory 71 . Supposing that accessing to the address ad is subsequently performed again and the data 
that is supplied to the computing element 74 from the blocking and bit-lowering circuit 62 is x ' 2, the output of the com- 
puting element 72 becomes 2, and the output of the computing element 74 becomes x • 1 + x ' 2; therefore, the output 
20 of the computing element 75 becomes (x1 + x2)/2, and this value is written into the address ad of the data memory 60. 
And, into the address ad of the frequency memory 61 , the value 2 that is the output of the computing element 72 is writ- 
ten. Supposing that accessing to the address ad is performed again and the data that is supplied to the computing ele- 
ment 64 from the blocking and bit-lowering circuit 62 is x ' 3, the value (x1 + x2 + x3)/3 is written into the address ad of 
the data memory 60, and the value 3 is written into the address ad of the frequency memory 61 . 
25 [01 21 ] In this way, the mean value of the pixel values of the attentional pixels related to the attentional bit modification 
pixels that are classified into each class is stored in the data memory 70. 

[0122] In the case where the pixel values that have been stored in the data memory 70 is stored in the memory 53 of 
Fig. 16 as the data for prediction, reading the pixel value that is regarded as the predictive data from the memory 54 is 
predicting the negated pixel, therefore, it is not needed to provide the pixel data producing circuit 55. As to the memory 
30 13 of Fig. 16, the coefficient set that is regarded as the predictive data and that has been produced by the learning 
apparatus of Fig. 12 is stored therein. 

[0123] By the way, in the above-mentioned case, the blocking and bit-lowering circuit 51 of Fig. 16 forms one block (a 
class tap and a predictive tap) with eight bit-modification pixels (pixel data) A, B, C, D, E, F, G and H that are neighboring 
the attentional bit modification pixel x' upwards, downwards, left and right of the spatial direction, as shown in Fig. 17; 

35 however, it is able to form a block such that the block includes also the thinned pixels that are neighboring the attentional 
bit modification pixel x 1 in the temporal direction. In this case, it is able to restore the attentional bit modification pixel x 
' such that it becomes closer to the original, because clustering and calculation of the equation (3) are performed con- 
sidering the bit modification pixels (pixel data) of the temporal direction. In this case, it is needed to form a block in like 
manner at also the time of learning. Besides, a block (a class tap and a predictive tap) can be composed of only thinned 

40 pixels that are located in the temporal direction of the attentional bit modification pixel x. 

[0124] In this third embodiment, in the case where the attention has been paid to a certain attentional bit-modification 
pixel, the bit modification pixels that are used for clustering and the bit modification pixels that are used for calculating 
the linear expression shown in the equation (3) are identical; however, these are not always required to be identical. 
That is, it is able to use the distinct sets of thinned pixels, for clustering and for calculation of the equation (3). Besides, 

45 the method for clipping a class tap or a predictive tap may be changed adaptively, in accordance with the spatial feature 
(activity), movement, and others of the image. 

[0125] Besides, bit modification pixels that are used for clustering and bit modification pixels that are used for calcu- 
lating the linear expression shown in the equation (3) are not always required to neighbor the attentional bit modification 
pixel spatially or temporally. But, it is desirable to use the bit modification pixels that are resting on the periphery of the 

50 attentional bit modification pixel. 

[0126] Besides, ADRC processing has been used in the clustering circuit, and the class has been determined on the 
basis of the spatial activity; however, the movement of the block that has been detected may be used as the class, or 
the class may be determined based on the combination of ADRC processing and movement detection. 
[0127] In this embodiment, each pixel of the respective frames that compose a dynamic image was 12 bits, however, 

55 we do not intend to limit the present invention to such; each pixel may be comprised of the different number of bits, such 
as 8 bits, 1 0 bits and 1 6 bits. 

[0128] In this embodiment, the attentional bit modification pixel x ' (pixel data) is not included in the class tap or the 
predictive tap, however, we do not intend to limit the present invention to such; the attentional bit modification pixel (pixel 
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data) may be included in the class tap or the predictive tap. In this case, at also the time when the predictive data is 
produced for each class, it is needed that the class tap and the predictive tap include the attentional bit modification 
pixel (pixel data) similarly. 

[0129] Modifications of the succeeding circuits (the blocking circuit 1 1 , the clustering circuit 12, the memory 13, the 
5 interpolation data producing circuit 14 and the synthesis circuit 10) for processing the image data that is delivered from 
the image memory 56 are similar to those of the first embodiment, so the description is omitted. 
[0130] The transmitting/receiving system to which the invention has been applied has been described hereinbefore; 
such a transmitting/receiving system can be used in the case of recording/reproducing of image, as well as television 
broadcasting. 

10 [01 31 ] According to the image processing apparatus and the image processing method of the present invention, the 
pixels of the respective frames that compose the dynamic image are thinned in such a manner that they become the 
chessboard-like lattice pattern in both directions of the spatial direction and the temporal direction. Besides, according 
to the transmission medium and the transmitting method of the present invention, the data that is obtained by thinning 
the pixels of the respective frames that compose the dynamic image such that they become the chessboard-like lattice 

is pattern in both directions of the spatial direction and the temporal direction is transmitted. Therefore, it is able to reduce 
the quantity of information, maintaining the resolution of the dynamic image of horizontal, vertical and oblique direc- 
tions. 

[0132] Besides, according to the image processing apparatus and the image processing method of the present inven- 
tion, the original image is produced from the thinned pixels that is obtained by thinning the pixels of the respective 
20 frames composing the dynamic image in such a manner that they become the chessboard-like lattice pattern in both 
directions of the spatial direction and the temporal direction. Therefore, it is able to obtain the image whose deterioration 
of image quality is little. 

[0133] Besides, according to the image processing apparatus and the image processing method of the present inven- 
tion, the pixels of the respective frames that compose the dynamic image are thinned in such a manner that they 

25 become the chessboard-like lattice pattern in both directions of the spatial direction and the temporal direction, and, in 
addition, the bits of each pixel of the thinned pixel are subjected to bit-thinning. Besides, according to the transmission 
medium and the transmitting method of the present invention, the data is transmitted wherein said data is obtained by 
thinning the pixels of the respective frames that compose the dynamic image such that they become the chessboard- 
like lattice pattern in both directions of the spatial direction and the temporal direction and, in addition, exposing the bits 

30 of each pixel of the thinned pixels to bit-thinning. Therefore, it is able to reduce the quantity of information, maintaining 
the resolution of the dynamic image of horizontal, vertical and oblique directions. 

[0134] Besides, according to the image processing apparatus and the image processing method of the present inven- 
tion, the original image is produced from the thinned pixels that are obtained by thinning the pixels of the respective 
frames composing the dynamic image in such a manner that they become the chessboard-like lattice pattern in both 
35 directions of the spatial direction and the temporal direction and, in addition, exposing the bits of each pixel of the 
thinned pixels to bit-thinning. Therefore, it is able to obtain the image whose deterioration of image quality is little. 

Industrial Applicability 

40 [0135] A new image format is provided in which deterioration of image quality is little of the time when digital image 
data is transmitted and then the digital image data is restored. 

DESCRIPTION OF REFERENCE NUMERALS 

45 [0136] 1... input terminal, 2...subsampling circuit, 3.. .encoder, 4.. .transmission processing circuit, 5.. .output terminal, 
6.. .transmission line, 7... input terminal, 8.. .reception processing circuit, 9... decoder, 10...synthesis circuit, 1 1 ...blocking 
circuit, 12. ..clustering circuit, 13. ..memory, 14.. .interpolation data producing circuit, 15.. .output terminal, 21. ..input ter- 
minal, 22.. .blocking circuit, 23.. .data memory, 24.. .clustering circuit, 25.. .switch, 25a and 25b... terminal, 26.. .counter, 
27.. .arithmetic circuit of the method of least squares, 28.. .memory, 30.. .data memory, 31... frequency memory, 32 - 

so 35.. .computing element, 40... interpolation filter, 51 ...bit lowering circuit, 52.. .blocking circuit, 53... clustering circuit, 
54.. .memory, 55.. .pixel data producing circuit, 56.. .image memory, 10D... transmitter, 20D... receiver, 62.. .blocking and 
bit-lowering circuit, 63.. .data memory, 64.. .clustering circuit, 65.. .switch, 65a and 65b... terminal, 66.. .counter, 67.. .arith- 
metic circuit of the method of least squares, 68.. .memory, 70.. .data memory, 71 ...frequency memory, 72 - 75.. .comput- 
ing element. 

55 

Claims 

1. An image processing apparatus for processing an image data that composes a dynamic image, said image 
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processing apparatus comprises: 



thinning means for thinning the pixels of the respective frames that compose the dynamic image, in such a 
manner that they become a chessboard-like lattice format in both directions of the spatial direction and the 
5 temporal direction; and 

outputting means for outputting said thinned image data. 

2. The image processing apparatus as claimed in claim 1 , further including; 

10 bit-thinning means for performing bit-thinning of the direction of the level of the pixel, toward each of the pixels 

that compose said thinned image data, thereby producing a modified thinned image data; and 
said outputting means outputs said modified thinned image data. 

3. An image processing method for processing an image data that composes a dynamic image, said method com- 
15 prises the steps of: 

thinning the pixels of the respective frames that compose the dynamic image, in such a manner that they 
become a chessboard-like lattice format in both directions of the spatial direction and the temporal direction, 
thereby producing a thinned image data; and 
20 outputting said thinned image data. 



4. The image processing method as claimed in claim 3, further including the steps of: 

performing bit-thinning of the direction of the level of the pixel, toward each of the pixels that compose said 
25 thinned image data, thereby producing a modified thinned image data; and 

outputting said modified thinned image data. 



5. A transmitting method for transmitting the data that has been produced by thinning the pixels of the image data that 
composes a dynamic image, said method comprises the steps of: 

30 

thinning the pixels of the respective frames that compose the dynamic image, in such a manner that they 
become a chessboard-like lattice format in both directions of the spatial direction and the temporal direction, 
thereby producing a thinned image data; and 
transmitting said thinned image data. 

35 

6. The transmitting method as claimed in claim 5, further including the steps of: 



performing bit-thinning of the direction of the level of the pixel, toward each of the pixels that compose said 
thinned image data, thereby producing a modified thinned image data; and 
40 outputting said modified thinned image data as a transmission data. 

7. A transmission medium for use in transmission of a data that has been produced by thinning the pixels of an image 
data that composes a dynamic image, in which 

45 the data that is transmitted by said transmission medium is a thinned image data that has been produced by 

thinning the pixels of the respective frames that compose the dynamic image, in such a manner that they 
become a chessboard-like lattice format in both directions of the spatial direction and the temporal direction. 

8. The transmitting medium as claimed in claim 7, in which 

so 

a modified thinned image data, which has been produced by performing bit-thinning of each of the pixels that 
compose said thinned image data in the direction of the level of the pixel, is transmitted as said data. 

9. An image format of composing a dynamic image, in which 

55 

thinning the pixels of the respective frames that compose the dynamic image have been performed, in such a 
manner that they become a chessboard-like lattice format in both directions of the spatial direction and the 
temporal direction. 
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10. The image format as claimed in claim 9, in which 

with respect to each pixel of the data whose thinning have been performed for each frame, bit-thinning in the 
direction of the level of the pixel have been performed. 

5 

1 1 . An image processing apparatus for processing a thinned data that has been produced by thinning the pixels of the 
image data that composes a dynamic image, said image processing apparatus comprises: 

receiving means for receiving a thinned image data that has been produced by thinning the pixels of the 
io respective frames that compose a dynamic image, in such a manner that they become the chessboard-like lat- 

tice format in both directions of the spatial direction and the temporal direction; and 

restoring means for producing the negated pixels, with respect to said thinned image data, and restoring the 
original image data. 

15 1 2. The image processing apparatus as claimed in claim 1 1 , in which 

said restoring means is comprised of 

determining means for determining a stated class that represents the property of the attentional negated pixel 
of said thinned image data; and 
20 generating means for predicting the original pixel in response to said determined class, and for generating the 

original pixel data. 

13. The image processing apparatus as claimed in claim 12. in which 

25 said generating means includes a memory for storing a predictive data, for each class, that has been previ- 

ously produced for each class on the basis of learning by the use of the image data that composes a dynamic 
image in a standard manner, reads out a predictive data that corresponds to said determined class, and gen- 
erates the original pixel data on the basis of the predictive data that has been read out. 

30 14. The image processing apparatus as claimed in claim 1 3. in which 

said memory stores a set of predictive data for each class as said predictive data; and 
said generating means reads out a predictive coefficient set that corresponds to said determined class, and 
generates the original pixel data on the basis of the read predictive set and the plural bit-thinned pixels that are 
35 located at the periphery of said attentional bit-thinned pixel. 

15. The image processing apparatus as claimed in claim 13, in which 

said memory stores a pixel value for each class as said predictive data; and 
40 said generating means reads out a pixel value that corresponds to said determined class, and outputs the pixel 

value as the original pixel data. 

16. The image processing apparatus as claimed in claim 12, in which 

45 said determining means determines a class with respect to said attentional bit-thinned pixel, using the bit- 

thinned pixels that have been located in the spatial and/or temporal directions. 

17. An image processing apparatus for processing a thinned data that has been produced by thinning the pixels of the 
image data that composes a dynamic image, said image processing apparatus comprises: 

50 

receiving means for receiving a modified thinned image data that has been obtained by further performing bit- 
thinning of the direction of the level of the pixel, with respect to each pixel of the thinned image data that has 
been produced by thinning the pixels of the respective frames that compose a dynamic image, in such a man- 
ner that they become the chessboard-like lattice format in both directions of the spatial direction and the tem- 
55 poral direction; and 

a restoring means for producing the original pixel with respect to each bit-thinned pixel of said modified thinned 
image data, thereby producing the thinned image data, and for restoring the original image data from the 
thinned pixels of said thinned image data. 
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18. The image processing apparatus as claimed in claim 17, in which 

said restoring means is comprised of: 

first determining means for determining a stated class that represents the property of an attentional bit-thinned 
pixel of said bit-thinned image data; 

first generating means for predicting the original pixel in response to said determined class, thereby generating 
the original pixel data; 

second determining means for determining a stated class that represents the property of the attentional 
negated pixel of the thinned image data that is composed of said pixel data which has been generated by said 
first generating means; and 

second generating means for predicting the original pixel in response to said determined class, and for gener- 
ating the original pixel data. 

19. The image processing apparatus as claimed in claim 18, in which 

said first generating means includes a memory for storing a predictive data, for each class, that has been pre- 
viously produced for each class on the basis of learning by the use of the image data that composes a dynamic 
image in a standard manner, reads out the predictive data that corresponds to the class that has been deter- 
mined by said first determining means, and generates the original pixel data on the basis of the very predictive 
data that has been read out; and 

each of said second generating means includes a memory for storing a predictive data, for each class, that has 
been previously produced for each class on the basis of learning by the use of the image data that composes 
a dynamic image in a standard manner, reads out the predictive data that corresponds to the class that has 
been determined by said second determining means, and generates the original pixel data on the basis of the 
very predictive data that has been read out. 

20. The image processing apparatus as claimed in claim 19, in which 

the memory of said first generating means stores a predictive coefficient set for each class as said predictive 
data; 

said first generating means reads out a predictive coefficient set that corresponds to said determined class, 
and generates the original pixel data on the basis of the read predictive set and the plural bit-thinned pixels that 
are located at the periphery of said attentional bit-thinned pixel; 

the memory of said second generating means stores a predictive coefficient set for each class as said predic- 
tive data; and 

said second generating means reads out a predictive coefficient set that corresponds to said determined class, 
and generates the original pixel data on the basis of the read predictive set and the plural bit-thinned pixels that 
are located at the periphery of said attentional negated pixel. 

21. The image processing apparatus as claimed in claim 19, in which 

the memory of said first generating means stores a pixel value for each class as said predictive data; 

said first generating means reads out a pixel value that corresponds to said determined class, and outputs the 

pixel value as the original pixel data; 

the memory of said second generating means stores a pixel value for each class as said predictive data; and 
said second generating means reads out a pixel value that corresponds to said determined class, and outputs 
the pixel value as the original pixel data. 

22. The image processing apparatus as claimed in claim 18, in which 

said first determining means determines a class with respect to said attentional bit-thinned pixel, using the bit- 
thinned pixels that have been located in the spatial and/or temporal directions; and 

said second determining means determines a class with respect to said attentional negated pixel, using the bit- 
thinned pixels that have been located in the spatial and/or temporal directions. 
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